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ABSTRACT 

Project Educational Quality in Adult Literacy (EQuAL) was 
initiated to improve the quality of adult basic and literacy education (ABLE) 
services provided to adult learners in Pennsylvania. Project EQuAL's 
objectives were as follows: (1) set performance standards to facilitate 

delivery of quality ABLE programs to adult learners; (2) institute a system 
of continuous program improvement for adult education service providers; and 
(3) develop a coordinated infrastructure of policy, program development, 
training, and technical assistance to support delivery of high-quality ABLE. 
The following phases of Project EQuAL were implemented in 1994-1998: (1) 
alignment of Pennsylvania's process for monitoring ABLE programs with 
indicators of program quality; (2) training of staff from 10 Pilot 1 sites to 
form EQuAL improvement teams and conduct a problem-solving process to improve 
programs; (3) training of staff at 10 Pilot 2 sites and development of 
policies to support Project EQuAL’s assessment and program activities; (4) 
enhancement of the pilot sites and implementation of policies and planning; 
and (5) training of staff at 64 Wave 1 sites. By the end of phase 5, a number 
of objectives related to state policy formation, training and technical 
assistance, and quality improvement had been achieved. Descriptions of the 
Project EQuAL cohorts are appended. (Contains 19 exhibits.) (MN) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



I 

I 



<N 

r- 



Abt 



Abt Associates Inc. 



55 Wheeler Street 
Cambridge, Massachusetts 
02138-1168 

617 492-7100 telephone 
617 492-5219 facsimile 



Improving Programs 
Through Policy and 
Professional 
Development: Lessons 
Learned from Project 
Educational Quality in 
Adult Literacy 
(EQuAL) 

1999 



Hampden Square, Suite 600 
4800 Montgomery Lane 
Bethesda, Maryland 
20814-5341 

301 913-0500 telephone 
301 652-3618 facsimile 



Prepared by: 

Judith A. Alamprese 
Eric M. Stickney 



640 North LaSalle Street 
Suite 400 
Chicago, Illinois 
60610-3781 



Submitted to: 

Pennsylvania Department of Education 
Bureau of Adult Basic and Literacy 
Education 



312 867-4000 telephone 
312 867 -4200 facsimile 



PERMISSION TO REPRODUCE AND 
disseminate this material has 

BEEN GRANTED BY 



5 ? 

C o 

> ER1C 

XhmaamifiiTiiiiiiiii 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



BEST COPY AVAILABLE 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 



CENTER (ERIC) 

£^tfThis document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



j 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



- ' 




Preface 



The Pennsylvania Department of Education’s Bureau of Adult Basic and Literacy Education 
(ABLE) embarked on a multi-year initiative in 1994 to improve the quality of basic education 
services provided to adults in the state. This initiative, Project Educational Quality in Adult 
Literacy (EQuAL), was designed as a systemic intervention to institutionalize the process of 
continuous program improvement in adult basic and literacy programs. ABLE’s guiding 
assumption was that programs’ provision of quality services to adult learners would result in 
learners’ attainment of positive outcomes. Under Project EQuAL, ABLE has developed a 
coordinated set of policies as well as training and technical assistance to support local programs’ 
activities in adopting a problem-solving process to program improvement. 

As part of Project EQuAL, ABLE commissioned COSMOS Corporation and subsequently Abt 
Associates Inc. to assist in the development and assessment of the initiative. This report, 
prepared by Abt Associates, describes Project EQuAL’s activities during 1994-1998, the results 
from these activities, and recommendations for ABLE to consider in moving forward with 
Project EQuAL. 

Abt Associates Inc.’s staff conducted a number of activities to document the progress of Project 
EQuAL, including meetings with ABLE staff, telephone interviews and site visits to programs 
participating in Project EQuAL, and surveys to Project EQuAL coordinators and technical 
assistance providers. The authors of this report extend their thanks to everyone who provided 
information as part of Abt Associates’ data collection process. The authors are particularly 
grateful to Cheryl Keenan, ABLE’s director, without whose vision and support Project EQuAL 
would not have been possible. 

While we are thankful for the assistance provided by others, the authors alone are responsible 
for the contents of this report. 



Judith A. Alamprese 
Eric M. Stickney 

Abt Associates Inc. 
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Executive Summary 



Introduction 

Since Summer 1994, the Pennsylvania Department of Education’s Bureau of Adult Basic and 
Literacy Education (ABLE) has led a statewide initiative to improve the quality of basic 
education and literacy services provided to adult learners in the state. Prompted in part by the 
federal and state call for accountability, but also by ABLE’s desire to lead a large and diverse 
system of service providers in offering high quality educational services to adults, Project 
Educational Quality in Adult Literacy (EQuAL) has changed the ways in which training, 
technical assistance, and local program operations in adult education are carried out in 
Pennsylvania. The Project EQuAL initiative has involved the creation of policies to support new 
accountability requirements and programmatic changes, as well as the development and 
implementation of training and technical assistance to facilitate the use of new program 
procedures. As part of the development of Project EQuAL, ABLE commissioned COSMOS 
Corporation and subsequently Abt Associates Inc. to assist in creating, implementing, and 
documenting the activities undertaken as part of Project EQuAL. This report, prepared by Abt 
Associates, describes Project EQuAL’s activities during 1994-1998, the results from these 
activities, and recommendations for ABLE to consider in moving forward with Project EQuAL. 



Project EQuAL’s Objectives and Assumptions 

Project EQuAL was designed to meet the following objectives: 

• To set performance standards that would facilitate adult education service providers’ 
delivery of quality programs to adult learners; 

• To institute a system of continuous program improvement for adult education service 
providers; and 

• To develop a coordinated infrastructure of policy, program development, training, and 
technical assistance to support the delivery of high quality adult education services. 

The Bureau of Adult Basic and Literacy Education’s director and staff made a number of 
assumptions in designing an initiative to meet these objectives. The first assumption was that 
change is both a “top-down” and bottom-up” process that is interactive and reinforcing. While 
it was essential that ABLE lead the initiative, set policies guiding the use of a performance 
management system, and deploy resources that could support the system, the staff from local 
adult education service providers also had a critical role in developing materials and processes 
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for continuous program improvement and in determining the ways in which change could best 
be facilitated in their programs. 

A second assumption was that an infrastructure had to be developed that would enable programs 
to improve the quality of their services, which in turn would result in improved learner outcomes 
such as learning gains, program participation rates, and learners’ satisfaction with the services 
being provided. This infrastructure would need to involve policy, program development, and 
training and technical assistance. 



A final assumption that ABLE made, based on the experience of other state adult education 
programs that had carried out program improvement initiatives, was that change takes time and 
the long-term effects of program improvement— enhanced learner outcomes— would not be 
realized in the near term. A number of practices would need to be in place in programs before 
there would be documented evidence that learners from Project EQuAL programs performed 
better than those from programs not involved in program improvement. These practices were 
accurate learner assessment, data reporting, data analysis, and quality instruction. 



Approach to Program Improvement 

ABLE’s approach to a statewide program improvement process was to design an initiative that 
was multi-year, iterative in its development, and involved a variety of stakeholders. Because of 
the number and diversity of adult education service provides in the state, any process involving 
development, training, and technical assistance would have to be undertaken in stages. Thus 
Project EQuAL was designed to be a staged implementation process with a series of pilot 
activities, whereby increasing numbers of service providers participated in training and technical 
assistance related to program improvement. 

As part of ABLE’s desire to build upon other states’ experiences in creating an initiative, it 
procured the services of Judith Alamprese— originally from COSMOS Corporation and then 
from Abt Associates Inc.— to assist with the development of Project EQuAL. Ms. Alamprese 
had worked with California, Connecticut, and Oregon in assessing the effects of their statewide 
improvement initiatives and wanted to use the lessons from these states in Project EQuAL. 
ABLE’s staff and Ms. Alamprese thus designed Project EQuAL as an initiative that involved 
assessing the current state of practice of service providers, developing and pilot testing program 
improvement interventions, refining the interventions, and assisting the state’s adult education 
staff development cadre in transferring the development and refinement processes. In this way, 
ABLE worked with an external consultant in developing an overall initial plan, and involved key 
stakeholders in crafting the key elements of the plan as well as in designing the materials and 
training support to implement the plan. 
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Another component of Project EQuAL was the monitoring of related federal and state policies 
and requirements as Project EQuAL evolved in order to ensure that EQuAL’s activities reflected 
emerging issues in the state’s human services delivery system. Through the involvement of 
interagency members on Project EQuAL’s Advisory Committee and the state director of adult 
education’s participation in interagency committees and task forces, policies and technical 
assistance to support Project EQuAL were developed in light of emerging state priorities. 

ABLE’s approach to developing the program improvement initiative involved the following five 
stages of activities: . 1 

• Planning and Policy Development-Phase I: Consistent with the principle of having 
an ongoing monitoring system, EQuAL’s first set of activities was to revise the state’s 
process for monitoring basic skills and literacy programs to align it with the indicators 
of program quality. The procedures for setting performance standards for adult education 
programs also were developed, as was the conceptualization for the process of beginning 
a continuous improvement process with a set of pilot programs; 

• EQuAL Pilot 1/Performance Standards Data Collection: During the first year, staff 
from 10 Pilot 1 sites were trained to form EQuAL improvement teams and conduct a 
problem-solving process to analyze data and improve their programs. Initial assessment 
data also were collected for determining the state’s performance standards, and ABLE 
announced the inception of the EQuAL initiative; 

• Expansion of EQuAL Pilot/Policy Development— Phase 2: During EQuAL’s second 
year, 10 Pilot 2 sites were trained and there was further collection of assessment data for 
setting draft performance standards. ABLE began the development of policies for 
coordinating its National Literacy Act of 1991, Section 353 funding for special projects 
to support the assessment and program improvement activities underway with Project 
EQuAL; 

• Enhancement of Pilot Sites/Implementation of Policies and Planning— Phase 2: 

During the third year, ABLE began its plan for expanding programs’ participation in 
EQuAL beyond the pilot sites. A process for identifying the next “wave” of EQuAL 
sites was determined and training and materials for supporting EQuAL’s expansion were 
created. Additional special projects were funded to develop materials for facilitating 
program improvement and for reorganizing the state’s professional development system; 
and 

• “Going to Scale” with EQuAL/Implementation of Policies: During EQuAL’s fourth 
year, 64 Wave 1 sites were trained. Basic education and literacy service providers were 
required to address the state’s performance standards for adult education, and the 
reorganized professional development system provided training and technical assistance 
to the Wave 1 sites. 
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Using the key elements of this approach as a guiding framework, ABLE’s staff, external 
consultant, staff development cadre, and local service providers undertook a number of activities 
to build an infrastructure and environment for continuous program improvement in adult 
education. 



Project EQuAL Activities 

ABLE facilitated a number of activities related to the development of policy and program 

processes and materials as well as the delivery of training and technical assistance in carrying 

out Project EQuAL’s multi-stage initiative. These activities were of the following types: 

• Intervention processes and policies that ABLE developed to guide service providers in 
continuous program improvement; 

• Products that service providers could use in operating their programs; 

• Training and technical assistance to facilitate change in service provider operations; and 

• Monitoring and evaluation activities to track the progress of EQuAL and identify needed 
processes and products. 

Intervention Processes and Policies. The activities related to the development of processes and 

policies to support the EQuAL intervention were: 

• The revision of the state’s system for monitoring basic education and literacy programs; 

• The development of a design for EQuAL that included two phases of pilot sites from 
which lessons learned would guide the expansion of the number and types of programs 
participating in the initiative; 

• The setting of draft performance standards for adult basic and literacy education based 
on a sample of learners from the EQuAL Pilot programs; 

• The development of the Project EQuAL intervention, which was a problem-solving 
process that staff from basic education and literacy programs could use in identifying 
problems or questions about their programs, collecting and analyzing data about these 
problems, drawing conclusions about the possible causes of the problem, and identifying 
and implementing solutions. Local program staff were trained to work in teams using 
this process and to report their findings on a Data for Decision-Making Log, which was 
reviewed and analyzed. Feedback on the logs then was provided to the local teams to 
facilitate their continuous program improvement activities; 
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• The formation of an EQuAL Advisory Committee, which was comprised of key 
stakeholders from the basic skills and literacy service provider system as well as from 
vocational education, labor, and human services; 

• The creation of a self-assessment process for EQuAL sites to use in reviewing the 
strengths and weaknesses of their programs; 

• The coordination of projects funded under National Adult Literacy Act of 1991, Section 
353 funds to support the implementation of Project EQuAl; 

• The reduction of the number of the state’s professional development centers (PDCs) to 
six, in order to streamline the delivery of professional development services; and 

• The revision of the process for adult basic education and literacy programs to apply for 
funding for adult education services. 

Products. ABLE supported the development of a number of products to assist programs in 
meeting the performance standards and engaging in program improvement. These included: 1) 
an assessment manual; 2) a self-assessment checklist; 3) Data-for- Decision-Making Log; and 4) 
an Adult Learner Skills Competencies manual. 

Training and Technical Assistance. The state’s training and technical assistance system 
evolved over the first four years of EQuAL from a system that addressed a wide variety of topics 
and staff needs to a more focused delivery system of services. Over this period, the development 
of assistance activities shifted from the external consultant to the state’s cadre of staff 
development providers. 

Monitoring and Evaluation Activities. Throughout Project EQuAL, the external consultant 
collected data to monitor the implementation of the Pilot sites and to determine needs for 
technical assistance. During the fourth year of EQuAL, when the external consultant no longer 
provided training and technical assistance, she collected data on the implementation of Wave 1 
sites. These data were used to obtain feedback from sites about the processes developed as part 
of EQuAL and to provide information about the sequence of the change process. 



Outcomes from Project EQuAL 

By the end of the fourth year of Project EQuAL, a number of outcomes had been realized in 
terms of the development of state policy, the provision of training and technical assistance, and 
the quality of services being provided to adult learners. 

Policy. ABLE set draft standards regarding the performance of adult basic education and literacy 
programs, and during program year 1997-98 local programs had to report data regarding the 
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extent to which they had met the draft standards. ABLE’s use of Section 353 monies evolved 
over time and, by the end of 1997-1998, funding priorities had been targeted and coordinated so 
that activities undertaken with Section 353 monies resulted in products and technical assistance 
that built upon and reinforced one another. These activities were coordinated so that they 
supported the program improvement processes being facilitated through Project EQuAL. The 
state’s monitoring process for adult education and literacy programs also was aligned with 
program quality indicators and the state staff were more effective in reviewing programs and 
providing recommendations to service providers concerning their program improvement 
activities. In addition, the goal of continuous program improvement was an underlying theme 
in all of ABLE’s activities supporting local service providers. 

Training and Technical Assistance. Through Project EQuAL, a training and technical 
assistance infrastructure was built that emphasizes continuous program improvement, facilitates 
service providers’ access to staff development directly related to program improvement, and 
promotes problem-solving and the collection and analysis of data in managing programs. The 
technical assistance system also utilizes peer assistance in transferring practices and in 
supporting local service provider staff in assessing the quality of their programs as they perform 
activities to improve their services. 

Quality of Programs. As a result of Project EQuAL, service provider staff have begun to work 
in teams to analyze their program operations and to change the ways in which they provide 
services to learners. Key areas of program changes have been in recruitment and retention 
activities; the collection of assessment data; the organization of record-keeping systems to 
facilitate the use of data; and the delivery of instruction. As a result of these improvements, 
Project EQuAL participants have reported that the morale in their programs has improved, they 
have better methods for determining the solutions to problems, and they are learning to use 
learner assessment and other data to address the needs of their learners more effectively. 



Recommendations for Future Directions 

As ABLE moves forward in implementing Project EQuAL, the following activities are 
recommended to strengthen and enhance the system that has been developed: 



Program Performance Standards 

• Conduct a validation study of the learner outcome standards through the analysis of 
assessment data from a sample of learners, using the Literacy Pro system to determine 
the level of gain across subpopulations of learners from a variety of service providers. 
This study will provide information that can be used to determine appropriate standards 
for each subpopulation of learners for whom data are reported; 
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Continue to provide training in the administration of assessment instruments to 
incumbent and new staff. It is critical that reliable and valid data be collected by local 
programs, and the percentage of pre-post valid scores is the key indicator that there are 
quality data. Staff need to be reinforced in the appropriate use of instruments and in 
strategies for ensuring the collection of pre-post data for each learner; 

Conduct a survey of programs to determine the extent to which programs are referring 
learners to further education or training and/or unsubsidized jobs, and the processes they 
have for documenting these outcomes. This survey would provide baseline information 
for determining the range of service providers who are engaged in these types of 
activities as well as their capacity for collecting the required outcome data; and 

Conduct further training on the use of the management information system for accessing 
data for managing programs, particularly in the area of instruction. Staff need assistance 
in interpreting the data that can be retrieved using Literacy Pro, and on how to use these 
data at the class level for organizing instruction and for providing feedback and guidance 
to learners. 



Continuous Program Improvement 

• Refine the training in the problem-solving process to provide staff with skills in problem 
identification/question asking, data analysis and interpretation, and solution specification. 
Staff need assistance in determining how to specify a question about a program issue, 
methods for collecting new or examining existing information, and strategies for 
analyzing and interpreting data using basic data reduction techniques. Staff also need 
suggestions on how to develop feasible solutions to the problems that they have posed; 

• Gather information about the types of changes that programs have made in the key areas 
of program improvement and disseminate lessons learned about the change process. 
Through EQuAL, program staff have worked on a number of similar issues and made 
program changes based on their review and analysis of program information. It would 
be helpful for program staff to learn about the variety of changes that programs have 
made in the core areas of program operations that have been the focus of EQuAL’s 
problem-solving process. 

• Develop a strategy for assisting programs in improving curriculum and instruction— a key 
area programs identified as requiring change. As the federally funded research studies 
on ABE and ESL curriculum disseminate the results of their work, this information could 
be used to guide local programs in examining the quality of their instruction and 
curricular materials; and 

• Consider strategies for fostering the participation of program administrators in the 
improvement process and for addressing the need for time, staff, and fiscal resources in 
changing program operations. Program administrators are critical to the implementation 
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of continuous program improvement, and they need assistance in determining the ways 
in which they can provide their staff with time and resources to work together in 
examining the operation of their programs and implementing changes. 



Policy, Training, and Technical Assistance 

Policy. New policy issues that will need to be. considered are: 

• Incentives and strategies for programs to institutionalize and expand the improvement 
activities begun under Project EQuAL, including the provision of time, staff, and fiscal 
resources for planning, developing, and implementing program changes. Options need 
to be considered, such as reducing the number of learners served, so that there are 
resources for paying staff to engage in program improvement that will enhance the 
likelihood of success of the learners who are being served; 

• Refinements in the performance standards to reflect the needs of the various learner 
subpopulations and service provider systems. The performance standards need to be 
revised to reflect the new reporting requirements under the Adult Education and Family 
Literacy Act; 

• Incentives and strategies for programs to change their curriculum and assessment 
processes, which are essential to the teaching and learning process and are among the 
most difficult types of program improvements. Strategies such as supporting the 
evaluation of innovative practices at the local level might be considered as one step in 
developing a base of information for program improvement; 

• Strategies for providing extended learning through distance education and other methods 
to address the need for more time and intensity in instruction. One possibility is to 
designate pilot projects that would experiment with a range of existing distance learning 
tools (e.g., Crossroads Cafe, ALMA, Internet-based instruction developed by the Cyber 
STEP project) to determine methods for serving new populations of learners or extending 
the amount of instruction received by existing learners; and 

• Strategies for programs to expand or change the roles of staff to provide adequate 
capacity for collecting, analyzing, and using data for all aspects of program operations, 
and for fostering learner articulation with other education, job training, and support 
services in the community. One possibility is to train a core of staff regionally who could 
provide assistance to programs in the collection and analysis of data and who would 
receive periodic follow-up training to enhance the quality of their skills in these 
activities. 
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Training and Technical Assistance. Two critical areas in which technical assistance is needed 
are: 



Analyzing program issues, identifying appropriate data collection and analysis methods, 
and interpreting data results; and 

Providing feedback to staff in their activities in conducting program self-assessment and 
preparing Data for Decision-Making Logs. 



Workforce Investment Issues 

The Workforce Investment Act of 1998 poses a number of challenges to adult education in the 
unified plans that have been developed for the delivery of job training and basic skills. Issues 
that require attention are: 

• Strategies for promoting interagency collaboration whereby learners can access the 
variety of support services that they need to meet their educational goals. Local 
program staff may require training in strategies for collaborating and negotiating 
agreements with other providers who can offer additional services to learners 
participating in adult education; 

• Models for learner articulation so that learners are aware of and can access available 
education and job training services. To meet the requirements of the Workforce 
Investment Act, adult basic skills providers will need to participate in One-Stop systems 
to provide additional support to adult clients. Local adult education staff may require 
technical assistance in operating One-Stop services and strategies for participating in 
these services. The emerging national research on One-Stop systems would provide an 
information base from which technical assistance could be developed; and 

• Strategies for promoting a coherent basic skills and job training system that builds upon 
and expands the resources in a community. At the state level, ABLE will need to 
continue its interagency activities in building relationships and developing policies that 
can facilitate the operation of an integrated service delivery system at the local level. At 
the local level, basic skills providers will need to be trained on how to develop a strategy 
for assisting in the development of an integrated service delivery system, which includes 
assistance in conducting an environmental scan of the services in their community, 
collaborating with service providers, and working with learners to motivate them to 
access the range of services that will be available. 
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I. Pennsylvania’s Approach to Systemic Change in Adult Education 



Introduction 

Since Summer 1994, the Pennsylvania Department of Education’s Bureau of Adult Basic and 
Literacy Education (ABLE) has led a statewide initiative to improve the quality of basic 
education and literacy services provided to adult learners in the state. Prompted in part by the 
federal and state call for accountability, but also by ABLE’s desire to lead a large and diverse 
system of service providers in offering high quality education services to adults, Project 
Educational Quality in Adult Literacy (EQuAL) has changed the ways in which training, 
technical assistance, and local program operations in adult education are carried out in 
Pennsylvania. Through this initiative, state policies have been created to support new 
accountability requirements and training and technical assistance have been provided to improve 
the operation of basic education and literacy services. 

As part of the development of Project EQuAL, ABLE commissioned COSMOS Corporation and 
subsequently Abt Associates Inc. to assist in creating, implementing, and documenting Project 
EQuAL’s activities. This report, prepared by Abt Associates, describes ABLE’s approach in 
creating Project EQuAL, EQuAL’s activities that were undertaken during 1994-1998, the results 
from these activities, and recommendations for ABLE to consider in moving forward with 
Project EQuAL. 



Project EQuAL’s Objectives and Assumptions 

Context for Project EQuAL. ABLE’s main goal in establishing Project EQuAL was to begin 
a process of program improvement that would assist local adult education service providers in 
meeting new federal and state accountability requirements and in offering high quality literacy 
and basic education services to adults. Pennsylvania’s large and diverse system of over 200 adult 
education service providers, consisting of school districts, intermediate units, community 
colleges, and community-based organizations, varies from agencies serving thousands of learners 
in the state’s urban centers to smaller entities providing tutoring to adults in rural and urban 
settings. Prior to Project EQuAL, adult education service providers had been engaged in the 
process of defining program quality indicators and discussing possible ways of addressing the 
emerging accountability requirements. As the new director of ABLE considered the needs of 
Pennsylvania’s adult education system, she determined that the establishment of a statewide 
initiative would help to create an environment for change through the development of 
information and tools for assisting service providers in meeting new requirements and 
encouraging continuous improvement. The idea was to use the development of performance 
standards as a starting point for a statewide initiative that would involve the alignment of 
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program development, training, and technical assistance with program needs in order to support 
learner outcomes and quality services. 

Thus Project EQuAL was established with the following three objectives: 

• To set performance standards that would facilitate adult education service providers’ 
delivery of quality programs to adult learners; 

• To institute a system of continuous program improvement for adult education service 
providers; and 

• To develop a coordinated infrastructure of policy, program development, training, and 
technical assistance to support the delivery of high quality adult education services. 

Guiding Assumptions. The Bureau of Adult Basic and Literacy Education’s director and staff 
made a number of assumptions in designing an initiative to meet these objectives. The first 
assumption was that change involves both a “top down” and “bottom up” process that is 
interactive and reinforcing. In the case of Project EQuAL, a “top down” approach was defined 
as the development of policies and program requirements to promote and guide change in basic 
education and literacy services. Since one of EQuAL’s objectives was the implementation of 
performance standards, ABLE needed to set policies that delineated the standards and the 
requirements that local programs would have to meet as part of the standards. To ensure that 
programs had training and technical assistance resources to support their implementation of the 
standards, ABLE also had to provide a coherent array of materials and activities for program 
staff. To achieve this outcome, ABLE funded a coordinated set of projects with its National 
Literacy Act of 199 1 , Section 353 monies that supported the policy requirements that were being 
developed. 

To balance the “top down” approach, ABLE created a “bottom up” process whereby a subset of 
service providers participated in the development of Project EQuAL through their activities as 
EQuAL Pilot programs. Twenty Pilot programs were the first implementers of EQuAL and, as 
such, contributed to the iterative process that characterized the evolution of the initiative. 
Another aspect of the “bottom up” character of Project EQuAL was the participation of the 
regional staff development providers in the creation of training and related materials to support 
the implementation of EQuAL. 

ABLE’s second assumption was that an infrastructure had to be built that would enable programs 
to improve the quality of their services, which in turn would result in improved learner outcomes, 
such as learning gains, program participation rates, and learners’ satisfaction with the services 
being provided. ABLE’s staff believed that this infrastructure would be comprised of policies, 
program development, and training and technical assistance (“top down” and “bottom up” 
processes) that could facilitate a local program’s improvement activities. The framework that 
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specifies the components of Project EQuAL’s infrastructure is presented in Exhibit 1.1. As 
shown in the exhibit, ABLE’s staff assumed that EQuAL’s infrastructure would consist of a 
variety of program improvement activities that would lead to near-term changes in state policy 
in adult education as well as in the operational procedures of local adult education programs. 
Furthermore, the new policies and subsequent improvements in program operations would result 
in longer-term changes in programs as well as in outcomes for learners. 

hi creating a framework for the infrastructure, ABLE’s staff thought that policies and technical 
assistance related to the use of learner assessment and data collection would be critical to the 
implementation of program standards and the systematic improvement of local programs. Thus 
they assumed that many of the near-term changes would be in assessment policies and practices, 
which would enable problems in the longer-term to collect quality assessment data that would 
reflect the learning gains of program participants. 

A final assumption that ABLE’s staff made was that change takes time and the longer-term effects 
of program improvement — learning gains — would not be realized in the near-term. The 
experiences of states such as California and Connecticut that had implemented statewide initiatives 
provided some lessons to Pennsylvania in the development of EQuAL. ABLE’s staff wanted to 
benefit from the experience of these other state adult education programs in crafting their initiative 
so that EQuAL could progress at a reasonable rate. One lesson from the other states’ experiences 
concerned the types of practices that would have to be in place before longer-term gains could be 
realized and documented. These practices were accurate learner assessment, data reporting, data 
analysis, and quality instruction. Another lesson was about the types of technical assistance that 
would be needed to support local programs in refining their activities. In developing the framework 
for EQuAL’s infrastructure, ABLE’s staff both built upon and expanded the activities that other 
states had undertaken in designing their program improvement initiatives. 



Approach to Program Improvement 

ABLE’s approach to a statewide program improvement process was to design an initiative that 
was multi-year, iterative in its development, and involved a variety of stakeholders. Because of 
the number and diversity of adult education service providers in the state, any process involving 
development, training, and technical assistance would have to be undertaken in stages. Thus 
Project EQuAL was designed to be a staged implementation process with a series of pilot 
activities, whereby an increasing number of service providers participated in training and 
technical assistance related to program development. 

As part of ABLE’s desire to build upon other states’ experiences in creating an initiative, it 
procured the services of Judith Alamprese and her staff from COSMOS Corporation (who 
subsequently moved to Abt Associates Inc.), who had worked with California, Connecticut, and ' 
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Exhibit 1.1 

Project EQuAL Framework 
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Oregon in their program improvement initiatives. The role of the external consultant was to 
assist ABLE in designing and implementing a statewide initiative, including the processes for 
setting performance standards, the types of interventions needed to assist programs in improving 
the quality of their services, policies to support standards and program improvement, and the 
training, technical assistance, and data collection processes to facilitate and document the change 
process. 

Project EQuAL’s Strategy. In creating the strategy for Project EQuAL, ABLE’s director and 
staff wanted an initiative that could be a vehicle for refining the adult education system as well 
as for strengthening interagency relationships at the state and local level. While ABLE was 
focused on addressing the national call for accountability and performance standards in adult 
education, the staff also realized that this requirement was a part of the job training and social 
services systems providing services to many of the clients participating in adult education. 
ABLE’s intent was to equip adult education service providers with an accountability system that 
could assist clients in achieving their goals and in receiving related services. In many ways. 
Project EQuAL was conceived as an initial step in a long process of developing a coherent 
human services delivery system for undereducated and economically disadvantaged adults in 
Pennsylvania. 

ABLE’s strategy for developing Project EQuAL was to carry out a number of critical activities, 
assess their effectiveness, and design new activities that built upon the previous ones. In this 
iterative process, ABLE’s staff identified a number of principles that they wanted to guide 
EQuAL’s process. These principles were the following: 

• There should be an ongoing monitoring of programs and their practices to assure that 
programs are able to offer high quality services to adult learners; 

• Outside experts should be utilized in designing and implementing Project EQuAL to 
assure that EQuAL builds upon the lessons learned by other states and incorporates new 
information being developed through research and evaluation; 

• Key state and local stakeholders should be involved in the critical aspects of the design 
of EQuAL and in providing feedback concerning the implementation of EQuAL’s 
activities; 

• EQuAL’s interventions should be developmental and begin with a set of pilot programs 
whereby the interventions are evaluated, revised, and expanded to additional programs; 

• Products and training will need to be developed, and technical assistance provided to 
support service providers in their continuous improvement activities; 
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• Key change agents from local programs who are involved in the early implementation 
of Project EQuAL will be critical in assisting other programs in participating in EQuAL; 
and 

• Policies and processes will need to be continually reviewed to assure that they are 
adequate for guiding service providers and for addressing new requirements that may 
arise. 

At the heart of this strategy were two goals: to facilitate the operation of high quality adult 
education programs that, in turn, would assist adult learners in achieving outcomes that could 
improve the quality of their lives. ABLE’s strategy for Project EQuAL is depicted in Exhibit 1.2. 
As the exhibit indicates, EQuAL was conceived of as a series of sequential processes that would 
build upon one another and support the two main goals. ABLE’s staff also thought that these 
processes could be used in developing a number of specific programmatic interventions. For 
example, while an initial EQuAL activity was to develop performance standards, ABLE’s. staff 
recognized that other activities supporting program improvement would be needed and they 
wanted a strategy to use in implementing various interventions that would support continuous 
program improvement. 

Implementation Stages. When ABLE began Project EQuAL in the Summer 1994, the staff 
thought of it as a three-year initiative to implement performance standards in all basic education 
and literacy programs and to provide these programs with tools and materials to improve their 
programs. As the first two years of the initiative progressed, the complexity of facilitating change 
in the state’s diverse adult education system became apparent to ABLE staff, and the director 
revised the time line for EQuAL’s activities. Project EQuAL is now in its sixth year of 
implementation and continues to grow in scope and breadth. 

The first four years of EQuAL encompassed a variety of activities aimed at building an 
infrastructure for continuous improvement in all state-funded basic education and literacy 
programs. The stages of implementing Project EQuAL during this period, presented in Exhibit 
1.3, were the following: 

• Planning and Policy Development — Phase I: Consistent with the principle of having 
an ongoing monitoring system, EQuAL’s first set of activities was to revise the state’s 
process for monitoring basic skills and literacy programs to align it with the indicators 
of program quality. The procedures for setting, performance standards for adult education 
programs also were developed, as was the conceptualization for the process of beginning 
a continuous improvement process with a set of pilot programs; 

• EQuAL Pilot 1/Performance Standards Data Collection: During the first year, staff 
from 10 Pilot 1 sites were trained to form EQuAL improvement teams and conduct a 
problem-solving process to analyze data and improve their programs. Initial assessment 



Abt Associates Inc. 



Systemic Change in Adult Education 



1-6 



Exhibit 1.2 

Strategy for Implementing a State Initiative on Program Improvement 
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Assess first implementation based 
on experience and evaluation; 
refine processes and products; 
expand implementation 



Exhibit 1.3 

Project EQuAL: Stages of Implementation 
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data also were collected for determining the state’s performance standards, and ABLE announced 
the inception of the EQuAL initiative; 

• Expansion of EQuAL Pilot/Policy Development — Phase 2: During EQuAL’s second 
year, 10 Pilot 2 sites were trained and there was further collection of assessment data for 
setting draft performance standards. ABLE began the development of policies for 
coordinating its National Literacy Act of 1991, Section 353 funding for special projects 
to support the assessment and program improvement activities underway with Project 
EQuAL; 

• Enhancement of Pilot Sites/Implementation of Policies and Planning — Phase 2: 

During the third year, ABLE began its plan for expanding programs’ participation in 
EQuAL beyond the pilot sites. A process for identifying the next “wave” of EQuAL sites 
was determined and training and materials for supporting EQuAL’s expansion were 
created. Seven of the Pilot programs were designated as “core pilot sites” and enhanced 
their EQuAL activities by increasing the number of staff from their programs 
participating in the problem-solving process. Additional special projects were funded 
to develop materials for facilitating program improvement and for reorganizing the 
state’s professional development system; and 

• “Going to Scale” with EQuAL/Implementation of Policies: During EQuAL’s fourth 
year, 64 Wave 1 sites were trained. Basic education and literacy service providers were 
required to address the state’s performance standards for adult education, and the 
reorganized professional development system provided training and technical assistance 
to the Wave 1 sites. Abt Associates Inc. also evaluated the activities supporting the 
expansion of Project EQuAL. A list of the Pilot, Core, and Wave I sites is presented in 
Appendix A. 

These stages of implementation activities were developed with the idea that each stage required 
policy and program development as well as experimentation with local programs. Thus 
EQuAL’s approach as exemplified in its implementation strategy was to build an improved 
system by developing policies to facilitate change in local program activities that were supported 
by training and technical assistance. Over the four years, Project EQuAL became the focal point 
for ABLE’s activities in coordinating a systemic change process throughout the state’s adult 
education system. What began as an initiative to set performance standards and encourage 
continuous program improvement gradually emerged into a framework for organizing 
professional development and special project activities for fostering program improvement. 
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II. Project EQuAL Activities 



During the period 1994-1998, a variety of activities were carried out under Project EQuAL to 
support the development of performance standards and build an infrastructure for a continuous 
improvement system in adult education. These activities, which are presented in the timeline in 
Exhibit II. 1, were of the following types: 

• Intervention processes and policies that ABLE developed to guide service providers in 
continuous program improvement; 

• Products that service providers could use in operating their programs; 

• Training and technical assistance to facilitate change in service provider operations; and 

• Monitoring and evaluation activities to track the progress of EQuAL and identify needed 
processes and products. 



Intervention Processes and Policies 

Monitoring Review System. An initial step in setting performance standards was the refinement 
of the state’s system for monitoring basic education and literacy programs. The external consultant, 
working with ABLE’s director and staff, revised the process and materials that the state staff used 
in conducting monitoring visits annually to a sample of programs funded with National Literacy 
Act of 1991 monies. This activity included the alignment of program areas for monitoring under 
the state’s indicators of quality programs and the inclusion of systematic data collection tools for 
gathering information from instructors and a sample of learners. In keeping with Project EQUAL’S 
developmental strategy of pilot testing and revision, the external consultant trained the ABLE staff 
to use the new process and materials; collected data about their pilot testing of this process; and 
then refined the process based on the experience of the staff. A major goal of ABLE’s revised 
monitoring system was to reinforce to service providers the new emphasis on program 
accountability by having program staff focus on quality indicators in their review of their activities. 

Project Design. A key decision that ABLE’s director and staff had to make in designing Project 
EQuAL was the magnitude and sequencing of the initiative. Based on the lessons learned from 
other states, ABLE determined that EQuAL would begin with a pilot project of 10 basic skills 
and literacy service providers that was representative of the range of organizations and agencies 
sponsoring basic education and literacy services. The notion was to start small, learn from the 
experiences of the pilot sites, and then expand the number and types of programs participating 
in the initiative. Another consideration in selecting the first 10 Pilot sites was to choose 
programs 
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Exhibit 11.1 

Timeline of Key Project EQuAL Activities 
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interviews with Pilot land 
2 coordinators and 
program directors 



Exhibit 11.1 (Continued) 

Timeline of Key Project EQuAL Activities 
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where staff capacity could be developed for providing'training and technical assistance to 
subsequent EQuAL participants. Thus ABLE included in the Pilot 1 sites programs with 
directors and staff who had experience in developing special projects supported under the 
National Literacy Act of 1 99 1 's Section 353 funds and who were part of the state’s regional staff 
development system. The strategy with the Pilot 1 process was to both test new processes and 
develop capacity for transferring these processes. 

During the second year, 10 additional programs were selected as Pilot 2 sites. These programs 
served the same purposes as the Pilot 1 sites — providing assessment data for determining 
performance standards and participating in the Project EQuAL intervention described below. The 
Pilot 1 and 2 sites continued through the third year of the initiative, when seven of these sites were 
selected as Core Pilot sites and were asked to focus their activities on program improvement and 
expand the number of staff in their programs involved in EQuAL. 

The next phase of the design of Project EQuAL was to determine how to take the initiative to 
“scale.” During the third year, ABLE, the external consultant, and key staff from the Pilot sites 
developed a process for expanding the number of programs participating in EQuAL. During the 
fourth year, 64 basic skills and literacy service providers were selected through ABLE’s annual 
application process to become Wave 1 EQuAL sites. 

For each iteration of pilot and wave sites, the programs were asked to select a Project EQuAL 
team comprised of three to four staff members, including a program coordinator or staff member 
who had sufficient responsibility to effect change among the instructional staff. The objective 
of the EQuAL implementation team was to pilot the Project EQuAL intervention and to serve 
as an incubator for testing processes and products developed under EQuAL. The intent was for 
this team then to work with other staff members in institutionalizing Project EQuAL. 

To support the team’s activities, EQuAL sites were given a modest amount of funding each year 
($3,000-$5,000) for team meeting time and travel to EQuAL meetings. The idea was to 
stimulate the programs’ EQuAL activities by providing them with some additional funding. 
ABLE expected that the programs eventually would incorporate these activities into their 
ongoing program operations. During EQuAL’s fourth year, the additional federal monies that 
ABLE received for providing adult education services was used to fund the Wave 1 sites. 

Performance Standards. A key focus of Project EQuAL’s first year was designing the process 
for setting performance standards. The external consultant worked with ABLE’s director in 
developing this process based on the work that she had undertaken with Connecticut’s adult 
education system. This design was to collect learner assessment data on a sample of learners 
from the 10 Pilot 1 programs to determine: 1) the accuracy of the assessment data, and 2) the 
average gain made by learners. Since other states had found that adult basic education service 
providers often do not administer standardized assessment instruments in a reliable and valid 
manner, ABLE determined that the process of examining the quality of assessment data was 
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critical to the eventual implementation of standards. The approach of setting standards by 
examining the average gain made by learners was one of setting relative standards — i.e., 
standards based on the performance of learners relative to each other. While setting absolute 
standards — i.e., standards based on learners’ performance related to their achievement of a 
goal — would have been preferable, there were no data available about the long-term outcomes 
of participants from basic skills and literacy programs to use in determining the standards. Thus, 
ABLE chose to set the draft performance standards using learner data about relative performance. 

Both Pilot 1 and 2 sites collected assessment for setting the draft performance standards and the 
external consultant analyzed these data for ABLE. While the original design was to determine 
the draft standards based on the Pilot 1 sites’ assessment data, the number of valid pre- and post- 
test scores was insufficient for the analysis. Thus the data collection activity was extended to 
include the 10 Pilot 2 sites. The Pilot sites’ experience collecting assessment data was very 
informative and drew attention to an identified an area in which training and technical assistance 
was needed. During the third year, ABLE issued the draft performance standards that were then 
reviewed and reissued during the Fall 1998. 

EQuAL Intervention. Another part of the design activity was to determine the type of 
intervention that would constitute Project EQuAL. Because the long-term goal for EQuAL was 
continuous program improvement, a problem-solving process was created that staff from basic 
education and literacy programs could use in identifying problems or questions about their 
programs, collecting and analyzing about these problems, drawing conclusions about the possible 
causes of the problem, and identifying and trying solutions. The external consultant trained the 
Pilot site EQuAL teams to engage in a problem-solving process using a Data for Decision- 
Making Log (Exhibit II. 2). The goal of this process was to encourage staff to analyze their 
program operations using various types of data that programs collect, including intake 
information, attendance, learner assessment, instructional processes, and curricular materials. 
The teams were asked to meet monthly or every six weeks to work on a problem and to complete 
the log. Over the four-year period, the questions on the log were revised slightly to reflect the 
feedback from the Pilot sites. This problem-solving process was considered to be the basis for 
identifying program improvements that staff might make in an effort to offer quality educational 
services to adults. 

The EQuAL sites have utilized the Data for Decision-Making Logs with differing degrees of 
success. The external consultant provided technical assistance during the first two years of the 
initiative by conducting periodic telephone interviews with EQuAL coordinators at the Pilot sites 
to answer questions about the problem-solving process and provide suggestions. During the 
fourth year, the EQuAL Implementation team provided this assistance. At sites where the team 
has used the process effectively, team members have met regularly and carried out a systematic 
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Exhibit 11.2 



Project EQuAL 

1996-1997 

DATA FOR DECISION-MAKING LOG 

Name of pilot project site: 

Person(s) who completed this Log: 

Date of Completion: 



1 . Write a question about the program that you want to answer (e.g., curriculum 
and instruction, learner retention, support services, staff development). 

2. List the information that the program needs to answer the question. (The site 
must use data that the program currently collects or can easily collect to do 
the log.) 

3. Describe the analysis you used (e.g., frequency distribution, increase or 
decrease over time, pre and post). Please attach any charts, tables, or graphs 
that were used to summarize data. 

4. What conclusion(s) do you draw after analyzing the data? 

5. What changes, based on the analysis, might be suggested for the program? 

6. Who or what group of persons can make the decision to implement 
change(s)? 

7. List the resources you will need (e.g., release time, staff training, additional 
class space) and the steps you will take to carry out the changes (e.g., design 
new materials; hold training; revise procedures). 
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examination of one or more aspects of their programs to identify problems, solutions, and action 
steps. Factors facilitating this process have been leadership provided by the EQuAL coordinator, 
support from the program’s administrator in enabling staff to meet regularly, staff’s access to 
program information, and the staff’s capacity to take some steps in addressing program issues. 
At sites where the problem-solving process has been less successful, one or more of these 
elements has been missing. 



Advisory Committee. A key principle of EQuAL was to involve a variety of stakeholders early 
in the initiative. ABLE formed an EQuAL Committee comprised of representatives from the 
basic skills and literacy service provider system as well as from vocational education, labor, and 
human services. This Committee met two times during the first year and then ABLE’s director 
kept them apprised of EQuAL’s activities in subsequent years. The committee was helpful in 
reacting to the design for EQuAL and in early dissemination. 

Program Self-Assessment. During the third year of EQuAL, one of the Pilot 1 site directors 
who also conducted special projects and managed one of the regional staff development centers 
began the process of creating a self-assessment process for EQuAL sites. The intention was to 
have a process and instrument that staff could use in reviewing the strengths and weaknesses 
of their programs. This process was developed to complement the monitoring review system that 
the ABLE staff were using in their site reviews, as well as to stimulate effective problem-solving 
in the EQuAL implementation. In keeping with the EQuAL approach, the self-assessment 
instrument and process were pilot tested and revised. The instrument is a checklist that presents 
the key program areas related to the functioning of a basic education and literacy program with 
the types of activities that staff might undertake under each area. Staff are to rate the quality of 
their program activities in each area of operations. While programs have found the checklist 
helpful in focusing their discussions about program activities and areas in need of improvement, 
it also has taken time for staff to meet and rate each area of operation thoughtfully. 

This issue of staff time for meeting and discussing program operations is similar to that 
encountered in the use of the Data for Decision-Making Logs. While staff may value the activity 
of the systematic examination of their programs, they find it difficult to take the time to meet as 
a group, review their program activities, and then determine the steps that they might take to 
address the issues that they have identified. While one goal of Project EQuAL has been to 
institutionalize the staff’s activity of taking the time to analyze program problems, it has been 
challenging to get them to view this time as a necessary part of their program responsibilities. 

Alignment of Section 353-Funded Projects. An important policy activity that ABLE undertook 
was to coordinate the types of National Adult Literacy Act of 1991, Section 353 special projects 
that it funds. These monies generally supported program development, training, technical 
assistance, and evaluation activities. As EQuAL progressed, ABLE recognized the necessity to 
provide various tools to the pilot sites to facilitate their program improvement activities. In 
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addition, the process of setting draft performance standards pointed to the need for training and 
technical assistance in the administration and use of learner assessment instruments. 

As ABLE set its Section 353 priorities beginning in the second year of Project EQuAL, 
consideration was given to the issues concerning training and technical assistance that arose in 
the EQuAL implementation. To address these issues, ABLE specified projects that provided 
training for assessment and other identified skill-building related to staff competencies; product 
development to support going to scale with EQuAL; and technical assistance to facilitate EQuAL 
team activities. ABLE also funded projects such as the Literacy Pro management information 
system pilot implementation and practitioner inquiry projects that complemented the activities 
being undertaken with EQuAL. 

Professional Development System. One policy that ABLE developed during the third year of 
EQuAL was to reduce the number of professional development centers (PDCs) in the state to six. 
As part of its overall effort to reduce the number of providers in the adult education service 
delivery system, ABLE also wanted to streamline the delivery of professional development 
services. In reorganizing the PDCs, ABLE expanded their responsibilities to include the 
provision of technical assistance to the Wave 1 EQuAL sites that began operation during the Fall 
1997. 

ABLE Application Procedures and Program Guidelines. In the interest of promoting 
continuous program improvement, ABLE revised the application process for service providers 
to receive funding for adult education services. As part of the application process, programs now 
have to participate in the self-assessment process; submit data to address the performance 
standards; and participate in assessment training. 



Products 

ABLE has supported the development and revision of a number of products to assist programs 
in meeting the performance standards and engaging in program improvement. These have 
included: 

• An assessment manual that guides program staff in the selection and use of learner 
assessment instruments and processes; 

• Self-Assessment Checklist that programs can use in determining the areas of program 
operations that might need enhancement and those that are working well; 

• Data for Decision-Making Log that staff can use in guiding their discussions about 
program issues and problems, and in developing an action plan to address program 
issues; and 
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• Adult Learner Skills Competencies manual that lists core basic skills and life skills 
application areas. 

The first three products were developed as part of the Project EQuAL intervention activities for 
programs to use. The Skills Competencies manual was prepared as part of ABLE’s activity in 
providing programs with guidance for curriculum design. 



Training and Technical Assistance 

The state’s training and technical assistance system evolved over the first four years of EQuAL 
from a system that addressed a wide variety of topics and staff needs to a more focused delivery 
of services. With the implementation of EQuAL came the need for new training materials and 
processes as well as different content for technical assistance. The transformation of training and 
technical assistance followed the EQuAL strategy. The external consultant developed the early 
training and technical assistance materials, revised them based on the experience of the Pilot 
sites, and then worked with adult education training staff from the PDCs in transferring 
procedures that had been developed and in designing new materials and processes. During the 
third year, the external consultant assisted an EQuAL Working Group in developing a framework 
for training modules that could be used in going to scale or expanding the use of EQuAL in all 
basic skills and literacy programs. 

By the fourth year of EQuAL, the external consultant no longer delivered training. An EQuAL 
Implementation Team comprised of approximately 50 individuals from the PDCs became 
responsible for providing training and technical assistance to the 64 Wave 1 sites. The external 
consultant worked with the primary trainers from this group in reviewing their experiences and 
helping to determine the revision of training and technical assistance processes. 



Monitoring and Evaluation Activities 

Throughout Project EQuAL, the external consultant has collected data to monitor the 
implementation of the Pilot 1 and 2 sites and to determine the needs for technical assistance. 
During the fourth year of EQuAL, when the external consultant was no longer providing training 
and technical assistance, she worked with her staff in collecting data on the implementation of 
the Wave 1 sites. A summary of the data collection activities is presented in Exhibit II. 3: The 
results of these activities are discussed in the next section of the report. 

The ongoing collection of information from EQuAL sites has been an important part of the 
statewide initiative. One of its purposes has been to obtain feedback from sites about the 
processes developed as part of EQuAL, programs’ ease in undertaking change, and issues that 
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Exhibit 11.3 

Summary of Data Collection Activities 



Data Collection Activity 


Description 


Pilot 1 Telephone Interviews 


The 10 Pilot 1 sites began with Project EQuAL in 1994-95. Telephone 
interviews were conducted with all 10 sites in May 1995, January 1996, and 
May 1996. 


Pilot 2 Telephone Interviews 


The 10 Pilot 2 sites began with Project EQuAL in 1995-96. Telephone 
interviews were conducted with all 10 sites January 1996, and May 1996. 


Site Visits to Core Programs 


The Core programs are Pilot I and 2 sites that were selected to do additional 
implementation during 1996-97. Abt Associates staff conducted on-site 
interviews with staff from all 7 Core programs during 1997. 


Survey of Pilot 1 and 2 

Programs 

(1996) 


By 1996, two Pilot programs had ceased participating in EQuAL, leaving 18. 
Each of the 18 programs completed written surveys in May 1996. 


Analysis of Pilot 1 and 2 Log 

Questions 

(1995-1997) 


A review was conducted of the Pilot 1 and 2 sites’ Data for Decision-Making 
Logs completed from 1995 - 1997. This period represented the second and 
third years in Project EQuAL for the Pilot 1 sites, and the first and second 
years for Pilot 2 sites. 


Survey of Pilot 1 and 2 

Programs 

(1998) 


During 1998, another program left the project, leaving 17 Pilots, Each of the 
17 programs completed written surveys in May 1998. 


Survey of Wave 1 Programs 
(1998) 


The 64 programs that started with Project EQuAL in 1997-98 are known as 
Wave 1 sites. Surveys were mailed to the 64 Wave 1 sites in May 1998. Fifty- 
two returned surveys, representing an 81 percent response rate. 


Survey of Statewide 
Implementation Team 
(1998) 


The 23 members of the Statewide Implementation Team provide training and 
technical assistance to programs implementing Project EQuAL. Each of these 
individuals received a survey in July 1998. Thirteen Team members returned 
surveys, representing a 57 percent response rate. 
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ABLE needed to address in coordinating a systemic change process. The data collection also 
provided information about the sequence of the change process. For example, programs were 
given flexibility in selecting the program issues that they could choose to address in the Data for 
Decision-Making Logs. It was useful to have information that could trace the types of issues or 
problems that successive EQuAL sites addressed over time. Finally, the data collection provided 
some initial data about the near-term outcomes from Project EQuAL. 
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III. Outcomes from Project EQuAL 



During the first four years of Project EQuAL, the basic skills and literacy programs involved in 
the initiative focused on implementing a systematic process for identifying and solving problems 
that hindered the delivery of services. This approach was based on the assumption that the 
provision of quality services would result in improved outcomes for program participants. While 
good learner outcomes were thought to be the ultimate goal of quality services, ABLE 
recognized that, in the near-term, the effects of EQuAL would need to be assessed in terms of 
the extent to which programs were adopting a problem-solving process. This section describes 
the activities that EQuAL sites have undertaken toward implementing and institutionalizing this 
process. 



Assistance Provided by EQuAL Implementation Team 

Beginning in the fourth year of Project EQuAL, staff from the Professional Development Centers 
and the EQuAL Pilot sites became part of a statewide EQuAL Implementation Team that was 
responsible for providing training and technical assistance to the 64 Wave 1 programs that began 
EQuAL during 1997-98. During July 1998, Abt Associates’ research staff conducted a survey 
of the Implementation Team members to gather data on the types of assistance they had provided 
to Wave 1 sites, the barriers they experienced in providing this assistance, and the steps that 
might be taken to enable them to provide better technical assistance. Thirteen of the 23 
Implementation Team members completed a survey. 

The survey results indicated that the EQuAL Implementation Team members provided assistance 
to 34 (65 percent) of the Wave 1 sites. As indicated in Exhibit HI. 1, the most commonly reported 
types of assistance provided concerned the use of the Data for Decision-Making Logs and the 
application of technology, (e.g., help with the listserv, support with data entry/information 
management). These categories of assistance reflect the team members’ charge of assisting 
programs with the problem-solving process and helping them to meet ABLE’s requirements for 
learner accountability, including the use of a management information system. 

The main difficulties that Team members had in providing training and technical assistance to 
sites were related to the use of the Data for Decision-Making Logs. As shown in Exhibit III. 2, 
some Team members did not feel qualified to provide feedback on the logs to sites, primarily 
because of their lack of experience in analyzing logs and formulating advice based on the 
analysis. Other team members thought that the greatest challenge was assisting programs in 
using the log process as an instrument for change. 
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Exhibit 111.1 

Technical Assistance Received by Wave 1 Sites from 
Statewide Implementation Team 



Types of Technical Assistance 


Number and percent of 
Wave 1 Sites Reporting 
Having Received Technical 
Assistance 


Frequency Percent 

(n = 52 sites) 


No assistance received or requested 


1 8 34.6 


Assistance received, but no type specified 


1 3 25.0 


Assistance with data for decision-making logs 


9 17.3 


Assistance on technology-related issues 


9 17.3 


Assistance with the listserv 


5 9.6 


Assistance for data entry / information management 


oo 


Assistance with using computers 


1 1.9 


Assistance on purchase information 


3 5.8 



Data source: Survey of Wave 1 programs, Summer 1998 
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Exhibit 111.2 

Difficulties Encountered in Providing Technical Assistance and Training to Sites 
and Types of Professional Development Requested by Team Members 



Types of Difficulties Encountered by Team Members 


Frequency and percent of 
Team Members Reporting 


Frequency Percent 

(n = 13 team members) 


Team members did not feel qualified to provide assistance due to lack 
of experience with logs or adult education in general. 


4 30.8 


Sites needed more training and/or experience with the data for decision- 
making logs. 


4 30.8 


Team members had a lack of knowledge about Project EQuAL 
specifics and/or PDE procedures, and could not always answer sites’ 
questions. 


3 23.1 


Team members had a lack of time to adequately assist sites. 


3 23.1 


There was a lack of direct communication between team members and 
sites. 


2 15.4 


Types of Professional Development Requested by Team Members 




Data for decision-making log process, including question formation, 
data collection, data analysis 


7 53.8 


Sharing lessons learned of Implementation Team members 


3 23.1 


No professional development requested 


3 23.1 


Overview of the administrative and funding structures of adult 
education programs 


1 7.7 



Data source: Survey of Statewide Implementation Team, Summer 1998 
Respondents could give more than one answer 
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Eighty-five percent of the Team members responding to the survey cited the completion of logs 
as difficult for sites, and several pointed to the formation of a good research question as the 
major obstacle within the process. One Team member’s response seemed to capture this issue: 
“There are some sites who tried to attack all of their agency problems/issues at once instead of 
focusing on one issue or question. In other words, some sites presented material that could be 
addressed over several different logs.” Another wrote, “(Sites) have trouble asking questions 
that are answerable and that deal with small steps rather than huge, overarching issues.” Sites 
also had to address program barriers. Team members mentioned inadequate program leadership 
and logistical problems (i.e., getting staff to meet regularly) as some of these obstacles. 

When Team members were asked to identify a type of training that would be most helpful to 
them, more than half (54 percent) of the survey respondents requested assistance related to the 
log process, including question formation, data collection, and data analysis. Twenty- 
three percent of the respondents thought that it would be beneficial to share lessons learned with 
other Team members. Another 23 percent did not request any type of professional development. 
Presented in Exhibit III. 2 is a list of the types of training that Team members requested. 

\ 

Team members were asked what Project EQuAL could do to help programs succeed in making 
improvements. The survey respondents indicated that the Implementation Team should have 
increased interaction with EQuAL program staff. Suggestions included more face-to-face 
meetings, or regular phone or email conversations that could help program staff work through 
the log process. Other suggestions, included in Exhibit D3.3, were providing additional training 
for programs, giving programs more time to implement Project EQuAL in general, and offering 
more training for Implementation Team members on how to give feedback to programs on their 
Data for Decision-Making Logs. 



Pilot Site Outcomes 

Abt Associates’ research staff administered a survey to 17 Pilot 1 and Pilot 2 sites in the Spring 
of 1998 to determine the progress that these programs had made in using the problem-solving 
process and the changes that they had undertaken as a result. All 17 of the sites responded to the 
survey. The first section of the survey provided respondents with a list of the steps toward 
program change that they had indicated on their Data for Decision-Making forms during the 
1 996-97 program year. As Exhibit III. 4 indicates, most of the Pilot sites proposed that they 
would take six (35 percent of sites) or seven action steps (41 percent of sites). Respondents then 
were asked to indicate which of these steps they had fully or partially implemented and to 
describe the changes that their programs had made as a result of these activities. Among the 
programs that indicated they would take seven steps, six programs implemented six of the steps, 
while two programs implemented seven and four steps, respectively (Exhibit 113.4). Among the 
programs that proposed to take six steps, five implemented all six of those steps, while two 
programs implemented six and three steps, respectively. 
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Exhibit 111.3 

Suggestions from Statewide Implementation Team Members of Ways that 
Project EQuAL Can Help Programs Make Improvements 





Frequency and percent of 
Team Members’ Suggested 
Ways Project EQuAL Can 
Help Programs Make 
Improvements 


Suggested Ways Project EQuAL Can Help Programs 


Frequency Percent 

(n = 13 members) 


More interaction between Team members and program staff 


7 


53.8 


More training for program staff for data for decision-making logs, 
assessment modules, etc. 


5 


38.5 


More time for programs to move through the program improvement 
process 


2 


15.4 


More training for Team members regarding giving feedback to 
programs on their data for decision-making logs 


2 


15.4 



Data source: Survey of Statewide Implementation Team, Summer 1998 
Respondents could give more than one answer. 
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Exhibit III. 4 

Number of Action Steps Planned and Implemented 
by Pilot 1 and Pilot 2 Sites 



Number of Action Steps Planned by Pilot 1 
and Pilot 2 Sites 


Number and Percent of 
Pilot Sites 


Frequency Percent 

(n = 1 7 sites) 


7 action steps planned 


7 41.2 


6 action steps planned 


6 35.3 


5 action steps planned 


3 17.6 


3 action steps planned 


1 5.9 


Number of Steps Planned / 
Number of Steps Implemented 




7 steps proposed / 7 steps implemented 


1 5.9 


7 steps proposed / 6 steps implemented 


5 29.4 


7 steps proposed / 4 steps implemented 


1 5.9 


6 steps proposed / 6 steps implemented 


1 5.9 


6 steps proposed / 5 steps implemented 


4 23.5 


6 steps proposed / 3 steps implemented 


1 5.9 


5 steps proposed / 4 steps implemented 


2 1 1.8 


5 steps proposed / 3 steps implemented 


1 5.9 


3 steps proposed / 3 steps implemented 


1 5.9 



Data source: Survey of Pilot Sites, Spring 1998 
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The types of changes that programs undertook as a result of the problem-solving process can be 
categorized under eight areas of basic skills and literacy program activities, as shown in Exhibit 
m.5. Most of the Pilot sites made changes in program management, curriculum and instruction, 
and/or assessment. In program management, the most frequently cited activities were the 
development of record-keeping/ information management systems (47 percent) or the preparation 
of handbooks that are provided to learners and staff (18 percent). For example, one site 
explained, “We have continued to track our students for total gain. Reporting sheets for each 
class were designed for institutional implementation of this process.” 

In the area of curriculum and instruction, Pilot site staff reported upgrading instructional 
technology (24 percent), revising the curricula (24 percent), and/or implementing changes to 
class schedules (24 percent). Staff from one program observed, “Following the 12 hour staff 
development workshop covering computer literacy, slowly but surely computers are infiltrating 
each classroom.” Changes pertaining to assessment involved both the use of assessment 
instruments (35 percent) and the development of procedures (29 percent). Other broad categories 
into which program changes fell included staff development (47 percent), tutor issues 
(24 percent), learner retention (18 percent) and motivation (12 percent), and recruitment 
(12 percent). 

When asked to identify the factors that facilitated their implementation of change, Pilot site 
respondents cited high quality staff (88 percent of sites) and funding (76 percent of sites) 

(Exhibit III. 6). One respondent explained that “Staff and administration are both aggressive in 
new ideas and innovation for new things in the program” and another observed that “Fiscal 
resources and extra opportunities in staff training have enabled us to fully utilize our newly 
networked computers.” Respondents also noted that the hiring of additional staff, the provision 
of training, and the use of technology facilitated their program improvement activities. 

Pilot site staff also were asked to identify the factors that impeded the implementation of change 
in their programs. The two most common responses were time constraints of staff (59 percent) 
and the lack of funding (41 percent) (see Exhibit III.7). As one site explained, “No additional 
teacher hours were available to conduct extra math and English classes. No classroom space is 
available at our new location.” Other impediments included the need for additional training and 
the tendency of some staff to resist change. 

In order to understand further barriers to change, Pilot sites were asked to identify the factors that 
prevented the implementation of steps that were planned but not carried out, and the factors that 
would have enabled their implementation. Survey respondents cited constraints related to time 
(29 percent), funding (18 percent), and a lack of support from outside agencies (12 percent). 

Also, in 24 percent of programs, steps were not implemented simply because they were not a 
priority. One program explained, “So many personnel and programmatic changes occurred this 
year that follow through on these issues wasn’t the priority.” 
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Exhibit III. 5 

Changes in Program Operations Made by Pilot Sites 1 and 2 
as a Result of Log Activities Reported for 1996-97 





Number and Percent of Pilot 
Sites Reporting Implementing 
Changes to Areas of Program 
Operations * 


Areas of Program Operation 


Frequency Percent 

(n = 17 sites) 


Program Management 


10 


58.8 


Information management system 


8 


47.1 


Revised learner or staff handbook 


3 


17.6 


New staff hired 


1 5.9 


Implemented changes to attendance policies 


1 5.9 


New funding secured 


1 5.9 


Enlisted support from community agencies 


1 5.9 


Developing program improvement plan 


1 5.9 


Assessment 


8 


47.1 


Assessment instruments 


6 


35.3 


Assessment procedures 


5 


29.4 


Staff Development 


8 


47.1 


Curriculum and Instruction 


6 


35.3 


Upgraded instructional technology 


4 


23.5 


Implemented changes to curriculum 


4 


23.5 


Implemented changes to class schedule 


4 


23.5 


Tutor Issues 


4 


23.5 


Recruited and/or retained more tutors 


2 


1 1.8 


Tutor reporting system 


2 


1 1.8 


Provided training and assistance for tutors 


2 


1 1.8 


Learner Retention 


3 


17.6 


Learner Motivation 


2 


1 1.8 


Learner Recruitment 


2 


1 1.8 



Data source: Survey of Pilot Sites, Spring 1 998 

* All sites reported implementing at least one step; many sites reported more than one. 
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Exhibit III.S 

Factors Facilitating Pilot 1 and Pilot 2 Sites’ 
Implementation of Action Steps 



Factors Facilitating Implementation of Action Steps 


Number and Percent of 
Pilot Sites Reporting 
Factors Facilitating 
Implementation of Action 
Steps * 


Frequency Percent 

(n = 1 7 sites) 


High quality staff 


15 88.2 


Additional fiscal resources 


13 76.5 


Good administrative direction 


6 35.3 


Hired new staff 


3 17.6 


Provided training 


3 17.6 


Current technology 


2 1 1.8 


Support from community 


1 5.9 


Cooperative learners 


1 5.9 


Cooperative volunteers 


1 5.9 


Staffs organization of materials 


1 5.9 


Revised orientation 


1 5.9 


Implemented new testing instruments 


1 5.9 


Created attendance requirement 


1 5.9 


Program Improvement Team focusing on specific needs 


1 5.9 


Instructors given time to get used to new 
mate ri al s/proced u res 


1 5.9 


Increased cost of GED retakes 


1 5.9 


Sent information about program to support services 


1 5.9 



Data source: Survey of Pilot Sites, Spring 1 998 

* All sites reported at least one factor; many sites reported more than one. 
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Exhibit 111.7 

Factors Impeding Pilot 1 and Pilot 2 Sites’ 
Implementation of Action Steps 



Factors Impeding Implementation of Action Steps 


Number and Percent of 
Pilot Sites Reporting 
Factors Impeding 
Implementation of Action 

Steps * 


Frequency Percent 

(n = 17 sites) 


Time constraints of staff 


10 58.8 


Funding constraints 


7 41.2 


Need for additional training 


3 17.6 


Staff resistance to change 


2 1 1.8 


Confusion over standard practices 


1 5.9 


Uncooperative prison staff 


1 5.9 


Learner enrollment dropped 


1 5.9 


More paperwork involved 


1 5.9 


Need for outside consultant 


1 5.9 


Poor quality staff 


1 5.9 


Difficulty obtaining administrative approval 


1 5.9 


PDE made changes in assessment tools/procedures 


1 5.9 


.None 


2 1 1.8 



Data source: Survey of Pilot Sites, Spring 1998 

* All but two sites reported impeding factors; many sites gave more than one response. 
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The factors that would have enabled Pilot sites to carry out the steps were additional time 
(29 percent), additional funds (24 percent), and greater administrative support (18 percent). One 
program indicated, for example, that they needed “consistency and long term stability of 
programmatic direction.” 

Pilot sites also made changes to their programs that were not as a result of the log process. All 
but two pilot sites reported making these adjustments in program management and curriculum 
and instruction (see Exhibit 1H.8). Within program management, these activities were revising 
information management systems and program planning procedures. For curriculum and 
instruction, the most frequent modifications were in the overall curriculum (29 percent) and 
class schedule (29 percent), and the use of instructional technology (18 percent). Staff from one 
site, for example, observed that “Teachers began to collaboratively plan lessons across the 
curriculum areas in response to working on a mutual theme of career planning.” In addition, 
approximately a quarter (24 percent) of the Pilot sites made changes in their staff development 
activities, and 12 percent revised their assessment processes. At one site, they “Changed the 
GED schedule to 20 hours per week with open entry and exit to retain learners longer and gain 
goals more quickly.” 

Another item on the survey asked the Pilot sites to indicate whether their EQuAL teams still hold 
meetings, and if so, how often. About half of the sites (53 percent) continue to meet monthly and 
approximately one quarter of them (24 percent) noted that they meet “once or twice per month.” 
Two sites indicated that they meet biweekly and one site indicated that they meet weekly. Only 
one of the pilot sites indicated that their EQuAL team no longer meets. In spite of the lack of 
meetings, their site has undertaken several program improvement activities, including offering 
new training, maintaining positive staff relations, and searching for better curricular materials. 

Of the 16 Pilot sites that still hold EQuAL team meetings, only two have not added new staff 
members to the teams. At the sites at which teams still meet, new team members were usually 
instructors (59 percent) and administrators (53 percent). Other new team members included 
specialists (18 percent), volunteers (12 percent), support staff (12 percent), and a consultant 
(6 percent). 



Wave 1 Program Improvement Activities 

As a third type of data collection about the near-term outcomes from Project EQuAL, Abt 
Associates’ research staff conducted a survey of the 64 Wave 1 sites during May 1998. Fifty-two 
(8 1 percent) of the sites completed surveys. For the Wave 1 programs, learner retention was the 
most frequently discussed issue during 1997-98 (their first year in Project EQuAL). Sixty- 
four percent of Wave 1 program improvement teams focused on this topic, as shown in Exhibit 
III. 9 
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Exhibit IIL8 

Changes Made by Pilot 1 and Pilot 2 Sites 
in Addition to Action Steps Made Through the Log Process 



Changes Made in Addition to Action Steps 


Number and Percent of Pilot 
Sites Reporting 
Implementing Changes in 
Addition to Those Made 
Through the Log Process * 


Frequency Percent 

(n = 1 7 sites) 


Program Management 


10 58.8 


Information management system 


3 1 7.6 


Revised program planning procedures 


2 1 1.8 


Improved communication between staff members 


2 11.8 


New staff hired 


2 11.8 


Addressed use of IEPs 


2 11.8 


Stronger relations with PDE 


1 5.9 


Updated mission statement 


1 5.9 


Revised program directory 


1 5.9 


Curriculum and Instruction 


6 35.3 


Implemented changes to curriculum 


5 29.4 


Implemented changes to class schedule 


5 29.4 


Upgraded instructional technology 


3 1 7.6 


Use of new instructional resources 


1 5.9 


Staff Development 


4 23.5 


Assessment 


2 1 1.8 


Assessment instruments 


1 5.9 


Assessment procedures 


1 5.9 


Learner Recruitment 


2 1 1.8 


Tutor Issues 


1 5.9 


Learner Retention 


1 5.9 


Welfare Reform Issues 


1 5.9 



Data source: Survey of Pilot Sites, Spring 1998 

* All but two sites reported implementing measures in addition to the action steps implemented through 
the log process. Some sites reported more than one type of action. 
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Exhibit III. 9 

Areas of Program Operation Addressed by Wave 1 Sites 
During Their First Year of Participation in Project EQuAL 
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Exhibit 111.9 (Continued) 

Areas of Program Operation Addressed by Wave 1 Sites 
During Their First Year of Participation in Project EQuAL 
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Survey of Wave 1 Programs, Spring-Summer 1998. 



Learner retention was followed by curriculum and instruction as an area where changed was 
needed. Almost half of the Wave 1 programs specified topics such as the selection and 
development of curriculum/instructional materials, the use of classroom instructional strategies, 
the use of instructional technology, and the consideration of different learning styles. Other 
topics included assessment (44 percent), learner recruitment (39 percent), and program 
management (31 percent). For this third topic, respondents indicated that they were concerned 
primarily with information collection and data management, as well as with issues such as 
program evaluation, communication among staff, and program entrance and attendance 
requirements. Twenty-nine percent of the survey respondents noted concern about the ways in 
which programs interact with learners, including communications with and feedback from 
learners on topics such as learner achievement and learner goal-setting. 

Exhibit m.9 also presents a comparison of the topics discussed by the Pilot sites during their first 
year in Project EQuAL with those selected by Wave 1 sites during their first year (1997-1998). 
While the main areas of concern were the same (curriculum and instruction, learner retention, 
and assessment), their order of importance differed among the three cohorts of EQuAL sites. 
While curriculum and instruction was the area most frequently discussed by Pilot 1 sites, learner 
retention was of most concern for the Pilot 2 and Wave 1 sites. Assessment was the third most 
important issue across all three cohorts (about 40 percent of respondents). 

When Wave 1 programs were asked to identify steps they had taken toward program 
improvement during 1997-98, 39 percent reported that they had made changes in the way 
learners are assessed (see Exhibit III. 10). This most likely reflects the emphasis ABLE has put 
on program accountability and the availability of training in assessment. Other areas in which 
Wave 1 programs adjusted their operations were in curriculum and instruction (29 percent), 
program management (25 percent), interaction with learners (23 percent), and learner retention 
(21 percent). In each of these areas, a greater percentage of the survey respondents indicated 
that they had implemented activities than had specified the topic as one for discussion in the 
EQuAL team meetings. 

Each of the Wave 1 programs responding to the survey indicated that they would continue with 
the program improvement process, and named at least one topic or activity that they would be 
their focus. Presented in Exhibit III. 1 1 is a list of the topics they indicated they would address. 

Once again, curriculum and instruction as well as assessment were listed by over one third of the 
respondents. They were least likely to plan to focus on learner recruitment and tutor issues, 
which were named by just 14 and 9 percent of the respondents, respectively. 

When asked about the benefits from participating in Project EQuAL, almost all (96 percent) of 
the Wave 1 survey respondents listed a personal, staff, or program benefit. Only four percent did 
not list any benefits. Exhibit III. 12 shows the types of benefits reported by Wave 1 programs 
during their first year. 
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Exhibit 111.10 

Areas of Program Operations Addressed by Wave 1 Sites 





Number and Percent 


Number and Percent 


Number and Percent 




of Topics Discussed 
by Wave 1 Sites * 


of Topics 
Implemented bv 
Wave 1 Sites 


of Wave 1 Sites 
Reporting 
Implementing 
Activities, by Topic ** 




Frequency 


Percent 


Frequency 


Percent 


Frequency 


Percent 


Areas of Program Operation 


(n = 173 topics) 


(n = 108 topics) 


(n= 52 sites) 


Curriculum and Instruction 


36 


20.8 


16 


14.8 


15 


28.8 


Curriculum and instructional 
materials 


12 


6.9 


8 


7.4 


8 


15.4 


Classroom instructional strategies 


10 


5.8 


4 


3.7 


4 


7.7 


Instructional technology 


6 


3.5 


1 0.9 


1 1.9 


Adapting to learning styles 


5 


2.9 


1 0.9 


1 1.9 


Development or refinement of 
IEP/ILP 


2 


1.2 


1 0.9 


I 1.9 


Class size and schedule 


1 0.6 


1 0.9 


1 1.9 


Retention of learners 


33 


19.1 


11 10.2 


11 21.2 


Assessment 


23 


13.3 


20 


18.5 


20 


38.5 


Recruitment of learners 


20 


1 1.6 


11 10.2 ■ 


11 21.2 


Program Management 


18 


10.4 


13 


12.0 


13 


25.0 


Information collection and 
management reporting 


7 


4.0 


10 


9.3 


10 


19.2 


Program planning and evaluation 


3 


1.7 


0 


0 


Communication among staff 


3 


1.7 


0 


0 


Roles of staff 


1 0.6 


0 


0 


Site location and safety 


1 0.6 


0 


0 


Program entrance & attendance 
requirements 


2 


1.2 


3 


2.8 


3 


5.8 


Job satisfaction 


1 0.6 


0 


0 


Interaction with Learners 


18 


10.4 


13 


12.0 


12 


23.1 


Communication with learners in 
general 


8 


4.6 


8 


7.4 


8 


15.4 


Learner achievement 


7 


4.0 


0 


0 


Learner motivation 


2 


1.2 


0 


0 


Learner goal-setting 


1 0.6 


2 


1.9 


2 


3.8 


Connecting learners with community 


1 


0.6 


3 


2.8 


3 


5.8 


resources 














Staff Development 


13 


7.5 


7 


6.5 


7 


13.5 


Tutor Issues 


7 


4.0 


8 


7.4 


6 


11.5 


Recruitment and retention of tutors 


5 


2.9 


2 


1.9 


2 


3.8 


Tutor reporting 


1 0.6 


3 


2.8 


3 


5.8 


Training and assistance for tutors 


1 0.6 


3 


2.8 


3 


5.8 


Other specifics not given 


5 


2.9 


9 


8.3 


9 


17.3 



Data source: Survey of Wave 1 programs, Summer 1998 

* All sites reported at least one topic discussed; many sites reported more than one. 

** All sites but one reported at least one topic implemented; many sites reported more than one. 
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Exhibit 111.11 

Areas of Program Operations that Wave 1 Sites Plan to Discuss in the Future 





Number and Percent of 
Topics Reported by 
Wave 1 Sites 


Number and Percent of 
Wave 1 Sites Planning 
to Discuss Each Topic* 




Frequency 


Percent 


Frequency 


Percent 


Areas of Program Operation 


(n = 1 1 


7 topics) 


(n =. 


52 sites) 


Curriculum and Instruction 


27 


23.1 


19 


36.5 


Curriculum and instructional materials 


12 


10.3 


12 


23.1 


Classroom instructional strategies 


8 


6.8 


8 


15.4 


Instructional technology 


2 


1.7 


2 


3.8 


Understanding & adapting to learning styles 


1 0.9 


1 1.9 


Development or refinement of IEP/ILP 


4 


3.4 


4 


7.7 


Assessment 


18 


15.4 


18 


34.6 


Other: Not Specified 


14 


12.0 


13 


25.0 


Team will meet to develop new plans 


9 


7.7 


9 


17.3 


Team will monitor progress of on-going efforts 


5 


4.3 


5 


9.6 


Interaction with Learners 


15 


12.8 


14 


26.9 


Communication with learners 


4 


3.4 


4 


7.7 


Learner achievement 


5 


4.3 


5 


9.6 


Learner motivation 


2 


1.7 


2 


3.8 


Learner goal -setting 


4 


3.4 


4 


7.7 


Retention 


12 


10.3 


12 


23.1 


Staff Development 


1 1 9.4 


11 21.2 


Program Management 


1 1 9,4 


11 21.2 


Information collection and management, reporting 


5 


4.3 


5 


9.6 


Program entrance requirements 


2 


1.7 


2 


3.8 


Recognition of achievement of learners, staff 


2 


1.7 


2 


3.8 


Advisory board 


1 0.9 


1 1.9 


Hiring new staff 


1 0.9 


1 1.9 


Recruitment 


7 


6.0 


7 


13.5 


Tutor Issues 


6 


5.1 


5 


9.6 


Recruitment and retention of tutors 


3 


2.6 


3 


5.8 


Tutor reporting 


2 


1.7 


2 


3.8 


Training and assistance for tutors 


1 0.9 


1 1.9 



Data source: Survey of Wave 1 programs, Summer 1998 

* All but six ot the 52 responding sites specified at least one topic that the program improvement team planned 
to address next, and some identified more than one topic. The other six sites said that they planned to continue 
the program improvement process, but didn’t specify the topics they would address. 
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Exhibit 111.12 

Reported Benefits to Wave 1 Sites from Participation in Project EQuAL 



Types of Benefits 


Number and Percent of 
Wave 1 Sites Reporting 
Benefits * 

Frequency Percent 

(n = 52 sites) 


Benefits that are to the program improvement team 


49 


94.2 


Team members have benefitted from increased communication within 
program and understanding the program as a whole 


33 


63.5 


Team members feel more involved / appreciated 


7 


13.5 


Team members have benefitted from the implementation of the plans 
and improvement of the program in general 


5 


9.6 


Team members have benefitted from EQuAL training 


4 


7.7 


Team members benefitted no specifics given 


3 


5.8 


Benefits that are primarily to the program 


35 


67.3 


More contact and communication among staff that led to everyone 
having a better understanding of the whole program 


21 


40.4 


Program has become better organized and structured for continual 
improvement 


9 


17.3 


Observed positive benefits to the program in general because of PIT 
plan implementation 


4 


7.7 


New methods employed instruction and assessment that improved the 
program 


1 


1.9 . 


Benefits that are primarily personal 


18 


34.6 


I have better job performance and/or satisfaction because of this process 


8 


15.4 


More contact and communication with colleagues from other programs / 
idea sharing among programs 


5 


. 9.6 


I have optimism / confidence now that the program can be improved 
through this process 


3 


5.8 


More contact with and better understanding of learners 


2 


3.8 


Personal benefits no specifics given 


2 


3.8 


No reported benefits 


2 


3.8 



Data source: Survey of Wave 1 programs, Summer 1998. 

* All ot the 52 responding sites reported at least one type of benefit; many sites reported more than one type 
of benefit. 
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In terms of personal, staff, and program benefits, improved communications was named as a 
major outcome from participation in Project EQuAL. Several respondents discussed the value 
of having staff work with one another in a structured process to collectively solve problems. For 
example, one respondent said, “What the EQuAL project has done is structure the process so that 
team members can develop and implement plans collectively and that decisions are based on real 
data.” Through their work in EQuAL program improvement teams, staff have examined and 
discussed various aspects of the operation of their programs. This has helped individuals move 
from focusing on one aspect of the program to understanding the overall operation of the service. 
This has particularly assisted instructors, who often are not part of these discussions and 
decisions, to become more involved in program improvement 

Another reported benefit is that programs are interacting more with each other — through the 
EQuAL listserv or at EQuAL workshops — and are discovering solutions to common problems 
without “reinventing the wheel.” Several Wave 1 respondents also noted that their programs are 
stronger because of the changes they implemented as a result of undertaking the Data for 
Decision-Making Log process. 

A major goal of Project EQuAL is to assist basic education and literacy programs begin the 
program improvement process. Of the Wave 1 respondents, 92 percent thought that their 
experience with Project EQuAL had helped them develop a vision for program improvement. 
The other eight percent stated that they already had a vision in place for program improvement. 

Members of the EQuAL Statewide Implementation Team, which provides technical assistance 
to Wave 1 sites, also were asked to describe any changes they witnessed in programs. All of the 
Team members felt that Project EQuAL had changed participating programs in the following 
three ways: (1) increasing teamwork and communication within programs and between 

programs; (2) creating a process in each program to analyze their needs and activities and make 
improvements based on data, not assumptions; and (3) increasing the emphasis on program 
improvement in general. The views expressed by the Wave 1 respondents about the effects of 
Project EQuAL along with the perceptions of the Implementation Team members indicate that 
EQuAL has been effective in prompting program staff to address operational issues in a 
systematic way and to develop some new ways of working together in order to offer quality 
services to adults. 
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IV. Conclusions and Recommendations 



The Bureau of Adult Basic and Literacy Education initiated Project EQuAL to accomplish three 
objectives: 1) set performance standards, 2) institute a system of continuous program 
improvement, and 3) develop a coordinated infrastructure of policy, program development, 
training, and technical assistance. After four years of the implementation of EQuAL, a number 
of changes in practice, policy, training, and technical assistance have been undertaken in 
Pennsylvania’s adult education system. Discussed in this section are the lessons that we have 
learned about the implementation of Project EQuAL. Also presented are recommendations for 
ABLE to consider as the remaining programs in the basic education and literacy system are 
trained to implement EQuAL, as existing EQuAL programs expand their activities in program 
improvement, and as the context for adult education evolves with the Workforce Investment Act 
of 1998. 



Program Performance Standards 

The process of setting program performance standards in administrative reporting, enrollment, 
and learner achievement in adult education served a number of purposes. First, it provided 
ABLE with the opportunity to evaluate the types of assessment instruments that programs were 
using and the quality of assessment data that they were collecting. Furthermore, it raised 
awareness among programs about the important role of assessment, the need for a management 
information system that can be accessed at the program level, and the ways in which these data 
can be used in program management and learner feedback. Through the standards setting 
process, issues concerning staff capacity to conduct learner assessment also were identified that 
resulted in the development of products for assisting staff in using assessment instruments. 

Issues to Consider. ABLE accomplished its objective of setting standards, with draft standards 
being issued in February 1997 and the final standards in September 1998 (with ABLE’s 
specification that the standards are likely to undergo further revision as they are applied.) As 
ABLE moves forward in implementing the standards, a number of issues will need to be 
addressed. . These issues, listed below, concern both the implementation of the existing 
standards, and the state and local programs’ capacities to meet the requirements under the Adult 
Education and Family Literacy Act: 

• The extent to which the expected level of performance for learners should vary according to 
subpopulation group (e.g., first-level learners, intermediate-level learners). Recent research 
suggests that first-level learners may not make as much progress as intermediate and pre- 
GED learners given the same amount of instructional intervention. If these 
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subpopulations of learners do vary in the amount of progress they should be expected to 
make, then the performance standards will need to take these differences into account; 

• The types of assessment instruments that programs can use for measuring and reporting 
learner progress (e.g., use of reading instruments for first-level learners). Recent research 
also indicates that general literacy measures such as the TABE and CASAS may not 
document well the progress of first-level learners participating in reading instruction. An 
alternative is to use reading tests for adults that assess the specific reading skills that are 
taught to first level learners. The performance of first-level learners will need to be 
examined to determine if reading tests are more appropriate measures for documenting the 
progress of first-level learners; 

• In addition to reporting on adults’ gains in basic skill and receipt of a high school diploma 
or its equivalent, the Adult Education and Family Literacy Act requires that states provide 
data on adults’ placement in, retention, or completion of postsecondary education, training, 
unsubsidized employment, or career advancement. The extent to which local programs are 
engaged in activities that lead to these outcomes as well as their capacity to document these 
outcomes needs to be examined; and 

• The flexibility of the management information system for producing information that 
programs can use in managing instruction. Local program staff will need data about the 
demographic characteristics of learners, their attendance, and their assessment scores that can 
be analyzed at the program and class levels for optimal use in learner and program 
management. The management information system needs to be examined to determine its 
capacity to produce the types of data and analytical reports that are useful to program staff. 



Recommendations. In light of these issues and those related to the overall implementation of 
performance standards, the following activities should be considered to strengthen the validity 
of the standards, the use of assessment data, and the capacity of local programs to meet the 
reporting requirements under the Adult Education and Family Literacy Act: 

• Conduct a validation study of the learner outcome standards through the analysis of 
assessment data from a sample of learners, using the Literacy Pro system to determine the 
level of gain across subpopulations of learners from a variety of service providers. This 
study will provide information that can be used to determine appropriate standards for each 
subpopulation of learners for whom data are reported; 

• Continue to provide training in the administration of assessment instruments to incumbent 
and new staff. It is critical that reliable and valid data be collected by local programs, and 
the percentage of pre-post valid scores is the key indicator that there are quality data. Staff 
need to be reinforced in the appropriate use of instruments and in strategies for ensuring the 
collection of pre-post data for each learner; 




Abt Associates Inc. 



BEST COPY AVAILABLE 

Conclusions and Recommendations IV-2 

62 



• Conduct a survey of programs to determine the extent to which programs are referring 
learners to further education or training and/or unsubsidized jobs and the processes they have 
for documenting these outcomes. This survey would provide baseline information for 
determining the range of service providers who are engaged in these types of activities as 
well as their capacity for collecting the required outcome data; and 

• Conduct further training on the use of the management information system for accessing data 
for managing programs, particularly in the area of instruction. Staff need assistance in 
interpreting the data that can be retrieved using Literacy Pro, and on how to use these data 
at the class level for organizing instruction and for providing feedback and direction to 
learners. 



Continuous Program Improvement 

EQuAL’s Impact. Through Project EQuAL, program staff learned new strategies for working 
as a team, for analyzing the operation of their programs through a problem-solving prpblem 
process, and for identifying and implementing solutions to program problems. Three key factors 
have influenced staffs’ capacity to undertake these activities: 1) quality of staff, 2) time to carry 
out activities, and 3) administrative support. 

In reporting on their success in implementing program improvements, staff from the Pilot 1 and 
2 sites noted that having high quality staff facilitated their capacity to analyze program problems, 
identify possible solutions, and make changes in the operations of their programs. Also having 
the time to undertake the problem-solving process and to implement the changes that they 
identified through this process was critical. Where program staff were not able to meet as a 
team or examine their programs, the lack of time was cited as a primary impediment. While the 
quality of staff and availability of time are important resources in the program improvement 
process, the support from the program’s administrator also is crucial. Without this support, staff 
are hampered in their ability to make the changes that they identify in their team meetings. 

The EQuAL sites had differing degrees of success in using the problem-solving process. For 
many staff, the process of specifying a focused question that could be answered was difficult. 
Often multiple questions were stated as one, or the question that was stated was too broad. 
Another challenge was gathering and analyzing data that related to the questions that had been 
posed. Sometimes appropriate data were not available, or data that did exist were not easily 
accessible. In spite of these difficulties, staff worked hard to understand the issues in their 
programs that were impeding success and to identify feasible actions that they could take to solve 
program problems. 

The use of the problem-solving process as the major intervention for Project EQuAL revealed 
a range of issues that staff think need to be addressed in their programs. Across all EQuAL sites, 
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learner recruitment, curriculum and instruction, assessment, and interaction with learners were 
named as areas for improvement. The Pilot sites also identified recruitment and retention of 
tutors and reporting of tutor data as important issues. 

In assessing the extent to which the problem-solving process was institutionalized and had 
effects on program operations, the programs reported that all but one of the 17 operating Pilot 
1 and 2 sites still meet as a team, with half of them meeting monthly and others more frequently. 
The Pilot sites also were asked whether they had implemented action steps based on the Data for 
Decision-Making Logs that they completed in their team meetings. Sixty-five percent of the sites 
had carried out all or the majority of the program changes they had identified. Interestingly, most 
of the changes they reported were in the area of program management, followed by assessment 
and staff development. While these areas are not the ones that programs had identified in 
completing their logs during their first year of EQuAL, perhaps they found that improvements 
in programs were needed in these areas to build a base for changes in other areas. 

Recommendations. The experiences of the programs that have implemented EQuAL to date 
point to a number of activities that ABLE might consider in fostering continuous program 
improvement. These are: 

• Refine the training in the problem-solving process to provide staff with skills in problem 
identification/question asking, data analysis and interpretation, and solution specification. 
Staff need assistance in determining how to specify a question about a program issue, 
methods for collecting new or examining existing information, and strategies for analyzing 
and interpreting data using basic data reduction techniques. Staff also need suggestions on 
how to develop feasible solutions to the problems that they have posed; 

• Gather information about the types of changes that programs have made in the key areas of 
program improvement and disseminate lessons learned about the change process. Through 
EQuAL, program staff have worked on a number of similar issues and made program 
changes based on their review and analysis of program information. It would be helpful for 
program staff to learn about the range of changes that programs have made in the core areas 
of program operations that have been the focus of EQuAL’s problem-solving process. 

• Develop a strategy for assisting programs in improving curriculum and instruction— a key 
area programs identified as requiring change. As the federally funded research studies on 
ABE and ESL curriculum disseminate the results of their work, this information could be 
used to guide local programs in examining the quality of their instruction and curricular 
materials; and 

• Consider strategies for fostering the participation of program administrators in the 
improvement process and for addressing the need for time, staff, and fiscal resources in 
changing program operations. Program administrators are critical to the implementation of 
continuous program improvement and need assistance in determining the ways in which they 
can provide their staff with time and resources to work together in examining the operation 
of their programs and in implementing changes. 
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Policy, Training, and Technical Assistance 

Policy. Through Project EQuAL, ABLE has built an infrastructure of new policies to strengthen 
program operations and accountability as well as a refined training and technical assistance 
capacity to support program improvement. The major policy directives that ABLE developed 
as a result of EQuAL were the following: 

• Revision of ABLE’s program monitoring process; 

• Specification of program performance standards; 

• Alignment of National Literacy Act of 1991, Section 353 special projects funding; 

• Revision of program funding guidelines; and 

• Specification of requirements for development of program improvement plans and 
participation in assessment training. 

These policy directives have prompted programs to address the issues of accountability and 
improvement, as well as provided staff with tools to assist them in these processes. As ABLE 
moves forward with Project EQuAL, new areas for policy will need to be considered. These are: 

• Incentives and strategies for programs to institutionalize and expand the improvement 
activities begun under Project EQuAL, including the provision of time, staff, and fiscal 
resources for planning, developing, and implementing program changes. Options need to be 
considered, such as reducing the number of learners served, so that there are resources for 
paying staff to engage in program improvement that will enhance the likelihood of success 
of the learners who are being served; 

• Refinements in the performance standards to reflect the needs of the various learner 
subpopulations and service provider systems. The performance standards need to be revised 
to reflect the new reporting requirements under the Adult Education and Family Literacy Act; 



• Incentives and strategies for programs to change their curriculum and assessment processes, 
which are essential to the teaching and learning process and are among the most difficult 
program improvements to make. Strategies such as supporting the evaluation of innovative 
practices at the local level might be considered as one step in developing a base of 
information for program improvement; 

• Strategies for providing extended learning through distance education and other methods to 
address the need for more time and intensity in instruction. One possibility is to designate 
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pilot projects that would experiment with a range of existing distance learning tools (e.g., 
Crossroads Cafe, ALMA, Internet-based instruction developed by the Cyber STEP project) 
to determine methods for serving new populations of learners or extending the amount of 
instruction received by existing learners; and 

• Strategies for programs to expand or change the roles of staff to provide adequate capacity 
for collecting, analyzing, and using data for all aspects of program operations, and for 
fostering learner articulation with other education, job training, and support services in the 
community. One possibility is to train a core of staff regionally who could provide assistance 
to programs in the collection and analysis of data and who would receive periodic follow-up 
training to enhance the quality of their skills in these activities. 

Training and Technical Assistance. Since the inception of Project EQuAL, the state’s training 
and technical assistance system in adult education has been reorganized and redefined. While 
there are fewer professional development centers, the role of these entities has become more 
focused on program improvement and the facilitation of staff change. As this transformation has 
taken place, staff delivering training and technical assistance related to Project EQuAL have 
identified two critical areas in which they need further assistance. These are: 

• Analyzing program issues, identifying appropriate data collection and analysis methods, and 
interpreting data results; and 

• Providing feedback to staff in their activities in conducting program self-assessment and 
preparing Data for Decision-Making Logs. 

These activities are at the heart of the change strategy developed under Project EQuAL, and staff 
need to be better equipped at providing peer assistance in carrying out these processes. Since 
these are difficult activities, it may take a variety of strategies to reinforce the skills that staff 
need to perform them. For example, the specialization of staff in providing assistance in areas 
such as data analysis and interpretation may enhance the quality of the assistance that currently 
is being provided to local program staff. The use of electronic communication in giving 
assistance also would facilitate the use of technical assistants who are not geographically 
proximic to the program staff with whom they are working. 

In discussions with local program staff on program performance standards and continuous 
program improvement, a number of topics for training were identified. These topics will need 
to be reviewed and appropriate training and technical assistance developed. Given the content 
of some of the areas, such as curriculum and instruction, an effective strategy may be to have 
technical assistance staff specialize in different topic areas to ensure that quality assistance is 
provided. In guiding the development of materials for training, ABLE should draw upon the 
most recent research and evaluation studies that are being undertaken nationally. 



BEST COPY AVAILABLE 



Abt Associates Inc. 



Conclusions and Recommendations 



IV-6 



Workforce Investment Act Issues 



The Workforce Investment Act of 1998 poses a number of challenges to adult education in the 
unified plans that have been developed for the delivery of job training and basic skills. Issues 
that require attention are the following: 

• Strategies for promoting interagency collaboration whereby learners can access the variety 
of support services that they need to facilitate the accomplishment of their educational goals. 
Local program staff may need training in strategies for collaboration and how to negotiate 
agreements with other providers who can offer additional services to learners participating 
in adult education; 

• Models for learner articulation so that learners are aware of and can access the further 
education and job training services that are available. To meet the requirements of the 
Workforce Investment Act, adult basic skills providers will need to participate in One-Stop 
systems to provide additional support to adult clients. Local adult education staff will need 
technical assistance concerning models for operating One-Stop services and strategies for 
how they can participate. The emerging national research on One-Stop systems would 
provide an information base from which technical assistance could be developed; and 

• Strategies for promoting a coherent basic skills and job training system that builds upon and 
expands the resources in a community. At the state level, ABLE will need to continue its 
interagency activities in building relationships and developing policies that can facilitate the 
operation of an integrated service delivery system at the local level. At the local level, basic 
skills providers will need to be trained on how to develop a strategy for assisting in the 
development of an integrated service delivery system, which includes assistance in 
conducting an environmental scan of the services in their community, collaborating with 
service providers, and working with learners to motivate them to access the range of services 
that will be available. 



A Final Note 

As a process for policy and programmatic change, Project EQuAL has accomplished a great deal 
in a relatively short period of time based on historical patterns of organizational change. This 
process has indeed been developmental, with interventions and policies being designed as the 
process progressed. While the quantitative count of changes that programs have made is 
substantial, the transformation of program staff in operating in teams, in serving in leadership 
roles in developing training, and in providing peer assistance is the real measure of the near-term 
effects of Project EQuAL. An important lesson learned thus far is that it takes three critical 
elements to engage in systemic change: 1) strong leadership on the part of the state staff leading 
the initiative; 2) quality materials and training; and 3) a willing audience of service providers to 
reflect on their practices and experiment with new processes. These three ingredients have been 
at the core of the Project EQuAL initiative. 
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Appendix A 

Project EQuAL Cohorts 



Pilot 1 Programs (10) 

Project EQuAL First Year: 1994-1995 

Altoona Community Education Center 
Central Intermediate Unit 10 County Vocational- 
Technical School 

Community Women’s Education Project 
Greater Pittsburgh Literacy Council 
Lancaster/Lebanon Intermediate Unit 13 



Norristown Area Achor School District 
Pennsylvania State University-Beaver Adult 
Literacy Action 
Rockview 

State Correctional Institution 
Scranton Council for Literacy Advance 
(SCOLA) 

Tri-County QIC 



Pilot 2 Programs (10) 

Project EQuAL First Year: 1995-1996 

ARIN Intermediate Unit 28 
Cumberland Valley School District 
Greater Johnstown Career and Technology 
Center 

Goodwill Industries of Pittsburgh 
Lehigh Carbon Community College 



Midstate Literacy Council 
Northampton Community College 
Northwest Tri-County Intermediate Unit 5 
Partners for English as a Second Language 
Tuscarora Intermediate Unit 1 1 Adult Education 
and Job Training Center 



Core Sites (Seven sites from Pilot 1 and 2 Programs): 1996-1997 



Community Women’s Education Project 
Goodwill Industries of Pittsburgh 
Lancaster/Lebanon Intermediate Unit 13 
Midstate Literacy Council 



Northwest Tri-County Intermediate Unit 5 
Tri-County OIC 

Tuscarora Intermediate Unit 1 1 Adult Education 
and Job Training Center 
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Wave 1 Programs (64) 

Project EQuAL First Year: 1997-1998 



Adult Literacy Center of the Lehigh Valley 
Adult Literacy Lawrence County 
Adult Literacty Program - Bayard Taylor 
Adult Literacy Training Program (Nanticoke) 
Allegheny Intermediate Unit 
Allentown Area School District 
Bedford County Literacy Council 
Bradford-Wyoming County Literacy Program 
Bucks County Correctional ABE Program 
Bucks County Volunteers in Teaching 
Alternatives 

Carbon County Area Vocational Technical 
School 

Catholic Charities Immigration & Refugee 
Center for Literacy (Philadelphia) 

Central Susquehanna Intermediate Unit 
Chester County OIC 
Community Action Southwest 
Community College of Beaver County 
Community Learning Center (Philadelphia) 
Crawford Central School Distict 
Delaware County Community College 
District 1 199C Training 
Employment & Training Inc. 

Erie Adult Learning Center 

Fayette County Community Agency 

Forbes Road East AVTS 

Greater Erie Community Action Committee 

Germantown Women’sJEducational Project 

Gertrude A. Barber Center 

Hispanic American Council 

Indian Valley Opportunity Center 

Indochinese Council 

Intermediate Unit 1 



Lackawanna Junior College 
Lincoln Intermediate Unit 12 
Literacy Council Reading-Berks 
Literacy Council of Venango County 
Literacy Program-Reading Area Community 
College 

Lutheran Settlement House 
Lycoming County Literacy Project 
Mansfield University 
Metro Career Center 
Midwestern Intermediate Unit 4 
Mifflin County Library Literacy Program 
Montgomery County Intermediate Unit 
Nazareth Area Adult Education Program 
New World Association 
Peirce Jr. College 
Penn College Expansion Grant 
Penncrest School District 
Pennsylvania School for the Deaf 
Pennsylvania State University-Beaver/Adult 
Literacy Action 
Perry County Literacy Council 
PIC of Westmoreland/Fayette 
Puerto Rican Organizing Committee 
School District of Philadelphia 
Schuylkill Intermediate Unit 29 
Somerset County AVTS 
State College ASD 
Temple University 
Waynesboro School District 
Williamsport Area School District 
Won Kwang English as a Second Language 
York County Area Vocational Technical School 
YWCA Adult Literacy Center 
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